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We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  Acquisition 
Research  Program: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics) 

•  Program  Executive  Officer  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Program  Manager,  Airborne,  Maritime  and  Fixed  Station  Joint  Tactical  Radio  System 


fn  Kuvfr.ii, i 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  -  i  - 


•  Program  Executive  Officer  Integrated  Warfare  Systems 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  &  Technology) 

•  Deputy  Assistant  Secretary  of  the  Navy  (Acquisition  &  Logistics  Management) 

•  Director,  Strategic  Systems  Programs  Office 

•  Deputy  Director,  Acquisition  Career  Management,  US  Army 

•  Defense  Business  Systems  Acquisition  Executive,  Business  Transformation  Agency 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department  of 
Energy 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  Symposium. 


James  B.  Greene,  Jr.  Keith  F.  Snider,  PhD 

Rear  Admiral,  U.S.  Navy  (Ret.)  Associate  Professor 


fn  Kuvfr.ii, i 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  -  ii  - 


Panel  19  - System-of-Systems  Acquisition: 
Concepts  and  Tools 


Thursday,  May  12,  2011 


11:15  a.m.  -  Chair:  Rear  Admiral  David  H.  Lewis,  USN,  Program  Executive  Officer,  Ships 
12:45  p.m. 

Capability  and  Development  Time  Trade-off  Analysis  in  Systems-of- 
Systems 

Muharrem  Mane  and  Daniel  DeLaurentis,  Purdue  University 

System-of-Systems  Acquisition:  Alignment  and  Collaboration 

Thomas  Huynh,  John  Osmundson,  and  Rene  Rendon,  NPS 

Using  Architecture  Tools  to  Reduce  the  Risk  in  SoS  Integration 

Chris  Piaszczyk,  Northrop  Grumman 


Rear  Admiral  David  H.  Lewis — Program  Executive  Officer  Ships.  Rear  Admiral  Lewis  is  responsible 
for  Navy  shipbuilding  for  surface  combatants,  amphibious  ships,  logistics  support  ships,  support  craft, 
and  related  foreign  military  sales. 

Born  at  Misawa  Air  Force  Base,  Japan,  Lewis  was  commissioned  in  1979  through  the  Navy  ROTC 
Program  at  the  University  of  Nebraska-Lincoln  with  a  Bachelor  of  Science  degree  in  Computer 
Science. 
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Berkeley  and  an  MS  and  PhD  in  physics  from  UCLA.  His  research  interests  include  uncertainty 
management  in  systems  engineering,  complex  systems  and  complexity  theory,  system  scaling, 
system-of-systems  engineering  and  architecting,  and  system-of-systems  acquisition.  Prior  to  joining 
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optimization.  During  his  23  years  at  Lockheed  Martin,  he  was  also  teaching  part-time  in  the 
departments  of  Physics  and  Mathematics  at  San  Jose  State  University.  Dr.  Huynh  is  a  member  of 
INCOSE. 

John  S.  Osmundson — Associate  Research  Professor,  Systems  Engineering  and  Information 
Sciences  Departments  (joint  appointment),  NPS.  Dr.  Osmundson  received  a  BS  in  physics  from 
Stanford  University  and  a  PhD  in  physics  from  the  University  of  Maryland.  His  research  interest  is 
applying  systems  engineering  and  computer  modeling  and  simulation  methodologies  to  the 
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Abstract 

System-of-systems  (SoS)  acquisition  research  has  identified  lack  of  alignment  and 
lack  of  collaboration  as  two  important  issues  leading  to  problems  in  SoS  acquisition. 
This  paper  captures  the  exploratory  work  toward  improving  alignment  between  and 
collaboration  among  the  individual  system  programs  in  the  development  of  a  SoS.  A 
collaborative  web-based  system  is  proposed,  on  which  personnel  of  all  programs 
associated  with  a  SoS  can  input  and  retrieve  information  required  to  align  the 
individual  programs.  The  overall  development  of  the  SoS  and  component  systems  is 
treated  as  a  critical-path  network  and  the  need  points  for  component  system  inputs 
are  identified  as  intermediate  milestones  requiring  SoS-component  system 
collaboration.  An  attraction  mechanism  to  effect  SoS  inter-program  collaboration  is 
incorporated  in  a  model  capturing  this  web-based  SoS  collaborative  system. 
Simulation  using  this  model  then  provides  results  to  establish  the  feasibility  of  such  a 
SoS  collaborative  system.  This  work  forms  a  basis  for  building  a  web-based  SoS 
collaborative  system  to  support  Department  of  Defense  SoS  acquisition  programs. 
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Introduction 

The  most  common  type  Department  of  Defense  (DoD)  systems  of  systems  (SoS) 
development  is  one  in  which  a  SoS  is  to  be  created  by  integrating  separately  developed 
systems — legacy  systems,  developmental  systems,  or  some  combination  of  both.  Research 
in  SoS  acquisition  has  identified  lack  of  alignment  and  lack  of  collaboration  as  two  important 
issues  leading  to  problems  in  SoS  acquisition.  By  lack  of  alignment  it  is  meant  a  system  is 
not  ready  for  its  integration  into  a  SoS,  or,  because  of  the  lack  of  the  front-end  SoS  systems 
engineering  (SE),  the  SoS  integration  discovers  that  the  system  does  not  meet  the 
performance  requirements  or  the  interface  requirements.  By  lack  of  collaboration  it  is  meant 
the  individual  system  programs  fail  to  work  with  each  other  to  achieve  the  goals  of  the  SoS 
program. 

SoS  acquisition  requires  the  availability  of  surrogates  of  component  systems  and 
later  the  “as  built”  component  systems  in  a  timely  manner  in  order  to  support  SoS  integration 
testing.  However,  the  acquisition  schedules  for  the  component  systems  are  typically 
developed  independently  of  the  SoS  development  schedule.  There  is  thus  no  assurance 
that  the  SoS  integration  testing  can  be  completed  as  planned,  resulting  in  the  SoS  schedule 
slip  and  associated  cost  overrun.  Even  when  the  schedules  are  aligned,  but  because  of  the 
lack  of  the  front-end  SoS  SE,  a  system,  during  the  SoS  integration,  may  not  meet  the 
performance  requirements  or  the  interface  requirements  or  there  may  be  misalignment  of 
resources  to  support  SoS  integration  testing,  such  as,  for  example,  the  absence  of 
component  system  experts  to  support  SoS  integration  testing. 

The  lack  of  alignment  is  related  not  only  to  the  front-end  SoS  SE  in  the  SoS 
acquisition,  but  also  to  the  lack  of  collaboration.  Collaboration  in  the  development  of  a  SoS 
is  multi-dimensional — between  DoD  system  program  offices,  between  contractors,  and 
between  DoD  program  offices  and  contractors.  “Inter-organizational  collaboration  has  been 
cited  as  a  critical  requirement  for  successful  outcomes;  and  for  those  agencies  struggling  to 
achieve  their  goals,  lack  of  inter-organizational  collaboration  has  been  cited  as  a  factor 
accounting  for  failure  (Kirschman  &  LaPorte,  2008).  Inter-organizational  collaboration 
requires  collaborative  capacity.  Mirroring  the  definition  of  collaborative  capacity  by  Hocevar 
et  al.  (2007),  collaborative  capacity  in  SoS  acquisition  is  defined  as  the  ability  of  individual 
system  programs  to  enter  into,  develop,  and  sustain  inter-system  programs  in  the  pursuit  of 
SoS  collective  outcomes.  Such  collaborative  capacity  is  needed,  in  addition  to  contracting 
structure  and  organizational  structures  (Rendon,  Huynh,  &  Osmundson.,  2010;  Huynh, 
Rendon,  &  Osmundson,  2010),  to  effect  resolution  of  the  SoS  acquisition  issues  raised  in 
(Osmundson  et  al.,  2007).  These  issues  are  initial  agreement,  SoS  control,  organizing, 
staffing,  team  building,  and  training  data  requirements,  interfaces,  risk  management  at  the 
SoS  level,  SoS  testing,  measures  of  effectiveness,  emergent  behavior. 

The  issues  addressed  in  this  research  are  not  just  the  ability  of  individual  system 
programs  to  “enter  into,  develop,  and  sustain  inter-systems  programs,”  but  also  the 
approach  to  and  mechanism  of  inducing  or  motivating  the  individual  system  programs  to 
develop  and  maintain  such  an  ability.  The  mechanism  is  intended  to  remove  barriers 
against  and  implement  factors  favorable  to  the  realization  of  collaborations  among  the 
individual  system  programs.  The  approach  proposed  in  this  work  to  bring  about 
collaboration  among  the  individual  system  programs  is  to  combine  this  mechanism  and  the 
implementation  of  a  front-end  SoS  SE  in  the  SoS  acquisition.  As  the  lack  of  alignment  is 
tied  to  both  the  lack  of  the  front-end  SoS  SE  in  the  SoS  acquisition  and  the  lack  of 
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collaboration,  the  collaboration  brought  about  by  this  approach  in  turn  aids  in  improving  the 
alignment  of  the  individual  system  programs. 

As  constrained  by  the  scope  of  this  paper,  the  front-end  SoS  SE  in  the  SoS 
acquisition  is  not  discussed  here.  Its  discussion  can  be  found  in  Huynh  et  al.  (2010).  A 
quantitative  analysis  of  the  benefits  of  having  the  front-end  SoS  SE  in  the  SoS  acquisition  in 
SoS  acquisition  is  currently  conducted  as  part  of  a  master’s  thesis  (Heng,  201 1 ).  This  paper 
is  focused  only  on  collaboration  among  the  individual  system  programs  as  it  is  related  to  the 
misalignment  issue. 

Enhancement  of  program  collaborations  might  include  re-organization  of  program 
structures,  creating  new  program  structures,  and  use  of  incentives.  These  techniques, 
however,  are  not  necessarily  the  only  means  to  effect  enhancement  of  program 
collaborations.  In  this  work,  the  key  idea  underlying  the  approach  proposed  here  is  the 
collaborative  behavior  observed  on  some  existing  web-based  systems.  That  is,  we  extend 
what  has  been  done  with  web-based  collaborative  systems  to  a  system  to  facilitate  the 
development  of  a  SoS  through  collaborative  behavior  from  the  individual  system  programs. 
The  web-based  system  concept  inspires  the  mechanism  proposed  in  this  research  for  inter¬ 
program  collaboration.  To  quantify  the  performance  of  the  inter-program  collaboration, 
modeling  and  simulation  (M&S)  is  employed,  incorporating  factors  that  directly  contribute  to 
and  barriers  that  prevent  the  enhancement  and  sustainment  of  collaboration  among  inter¬ 
system  programs. 

System-of-systems  (SoS)  modeling  and  simulation  has  recently  been  applied  to  the 
problem  of  engineering  SoSs  in  order  to  prevent  undesired  emergent  behavior  (Osmundson, 
2009a).  Example  SoSs  that  have  been  studied  are  the  collateralized  debt  obligation  market 
(Osmundson  et  al.,  2009b)  and  the  North  American  electric  power  grid  (Osmundson  et  al., 

2008) .  Theoretical  studies  of  these  SoSs  have  also  been  carried  out  to  validate  the  results 
from  the  modeling  and  simulation  work  (Huynh  &  Osmundson,  2008;  Huynh  &  Osmundson, 

2009) .  The  results  of  these  studies  indicate  that  SoS  modeling  and  simulation  can  be  used, 
at  least  in  some  cases,  to  predict  undesired  emergent  behavior  in  SoSs  that  consist  of 
engineered  systems  and  non-engineered  systems,  including  people,  and  to  identify  ways  to 
prevent  or  mitigate  undesired  behavior. 

Essentially,  to  deal  with  the  lack  of  alignment  and  collaboration  in  SoS  acquisition, 
we  recommend  that  a  SoS  acquisition  program  institute  an  overarching  front-end  SoS  SE  in 
the  SoS  acquisition  program  and  to  implement  an  approach  to  achieving  collaboration 
among  the  individual  system  programs. 

In  this  paper,  a  collaborative  web-based  system  is  proposed,  on  which  personnel  of 
all  programs  associated  with  a  SoS  can  input  and  retrieve  information  required  to  align  the 
individual  programs.  The  overall  development  of  the  SoS  and  component  systems  is 
treated  as  a  critical-path  network  and  the  need  points  for  component  system  inputs  are 
identified  as  intermediate  milestones  requiring  SoS-component  system  collaboration.  An 
attraction  mechanism  to  effect  SoS  inter-program  collaboration  is  incorporated  in  a  model 
capturing  this  web-based  SoS  collaborative  system.  Simulation  using  this  model  then 
provides  results  to  establish  the  feasibility  of  such  a  SoS  collaborative  system. 

Our  goals  in  this  paper  are  as  follows: 

■  Discuss  in  some  detail  some  existing  web-based  collaborative  systems; 
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■  Explain  our  exploratory  work  toward  improving  alignment  between  and 
collaboration  among  the  individual  system  programs  in  the  development  of  a 
system  of  systems;  and 

■  Elucidate  the  approach  proposed  in  this  research  for  achieving  collaboration 
among  the  individual  system  programs. 

The  rest  of  the  paper  is  organized  as  follows:  We  first  describe  and  explain  the  web- 
based  collaborative  systems;  then  we  discuss  modeling  and  simulation  of  the  web-based 
collaborative  systems;  next  we  continue  with  a  discussion  of  the  SoS  inter-program 
collaboration  approach;  and  finally  we  end  with  some  remarks. 

Web-Based  Collaborative  Systems 

The  Underlying  Idea  of  Web-Based  Collaborative  Systems 

Many  web-based  systems  are  based  on  what  is  known  as  network  effect: 

[A]  network  effect  (also  called  network  externality  or  demand-side  economies  of 
scale)  is  the  effect  that  one  user  of  a  good  or  service  has  on  the  value  of  that 
product  to  other  people.  When  network  effect  is  present,  the  value  of  a  product 
or  service  increases  as  more  people  use  it.  (“Welcome  to  Wikipedia,”  n.d.) 

When  the  network  effect  is  present,  the  value  of  the  system  to  customers  or  collaborators  is 
thus  dependent  on  the  number  of  customers  or  collaborators  already  using  the  system. 

Network  effects  become  significant  after  a  certain  number  of  people  have  subscribed 
to  the  system,  called  the  critical  mass.  At  the  critical  mass  point,  the  value  obtained  from  the 
good  or  service  is  greater  than  or  equal  to  the  price  paid  for  the  good  or  service.  Cost  also 
incurs  in  using  a  web-based.  Cost  could  be  payment  of  money  for  a  service  or  product,  time 
to  prepare  inputs  for  the  system,  time  spent  using  the  system  before  a  match  is  found,  or  a 
loss  associated  with  the  risk  of  using  the  system  such  as  not  receiving  goods  paid  for, 
receiving  incorrect  goods,  or  some  other  loss.  There  may  also  be  some  cost  associated 
with  attracting  the  participants.  At  the  critical  mass  point,  the  value  obtained  from  the 
system  is  greater  than  or  equal  to  the  cost  encountered  when  obtaining  the  good  or  service 
provided  by  the  system.  As  the  value  of  the  good  is  determined  by  the  user  base,  this 
implies  that  after  a  certain  number  of  people  have  subscribed  to  the  service  or  purchased 
the  good,  additional  people,  because  of  the  positive  value/cost  ratio,  will  subscribe  to  the 
service  or  purchase  the  good. 

Prior  to  reaching  the  critical  mass,  and  depending  on  the  system  type,  the  system 
must  attract  early  adopters  by  investment  capital,  incentives,  or  other  means.  In  the  interim, 
before  the  critical  mass  is  achieved,  some  early  adopters  may  drop  out  of  the  system 
because  of  lack  of  perceived  value,  while  others  join  the  system.  Thus,  the  success  of  a 
web-based  system  depends  on  achieving  a  critical  mass  of  subscribers  before  the 
effectiveness  of  attracting  additional  subscribers  to  the  system  is  exhausted. 

The  system  factors  that  determine  the  success  or  failure  of  a  web-based  system 
include  the  number  of  subscribers  or  participants  as  a  function  of  time;  the  factors  that 
attract  a  subscriber;  the  factors  that  cause  a  subscriber  to  leave  the  system;  the  value  of  the 
system’s  services  or  to  the  subscriber/participant;  and  the  cost  of  the  system’s  services  or 
products  to  the  subscriber/participant.  The  term  ‘participant’  will  be  used  exclusively 
hereafter,  as  the  individual  system  programs  are  ‘participants’,  although  in  a  strict  sense  the 
term  ‘subscriber’  more  properly  refers  to  someone  who  pays  for  a  service  while  a  participant 
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refers  to  a  person  who  invests  time  and  effort  to  obtain  a  product  or  service,  but  does  not 
pay  money  for  it.  It  is  assumed  that  a  participant  wants  to  find  a  match  in  the  system — the 
match  may  be  with  another  participant,  or  with  a  product  or  service  provided  by  the  system 
that  meets  the  participant’s  search  criteria.  Value  to  the  participant  is  associated  with  finding 
a  match. 

Examples  of  Collaborative  Systems 

The  type  of  web-based  system  of  most  interest  is  a  collaborative  enterprise  whose 
success  depends  on  the  number  and  quality  of  the  participants,  but  not  on  how  much 
revenue  the  system  attracts.  Examples  of  this  type  of  system  are  those  that  are  established 
to  facilitate  a  process  through  collaborative  behavior  such  as  eBay,  Facebook,  and  the 
Zerox  Eureka  system. 

eBay  is  an  online  auction  and  shopping  website  in  which  individuals  and  businesses 
buy  and  sell  a  wide  variety  of  products  and  services.  eBay  was  founded  in  1995  and 
experienced  very  rapid  growth.  By  the  second  year  of  operations  eBay  hosted  250,000  on¬ 
line  auctions,  and  2  million  on-line  auctions  the  following  year  (“eBay,”  n.d.).  Facebook  is  a 
social  networking  website  that  began  in  February  2004  and  had  more  than  500  million 
participants  by  July,  2010  (“Statistics,”  n.d.).  Participants  maintain  personal  profiles,  can 
add  people  as  friends,  send  messages  to  friends,  notify  friends  about  updates  to  their 
profile,  and  access  friends’  profiles.  The  Eureka  system,  developed  by  Xerox  (“The  Eureka 
Project,”  2010),  allows  customer  service  engineers  to  share  validated  tips  on  problems 
encountered  and  solutions  on  Xerox’s  family  of  copier  machines.  The  system  is  an  example 
of  a  net-based  community  of  practice  within  an  organization.  Customer  service  engineers 
browse  the  Eureka  system  to  see  if  there  is  a  known  solution  to  a  problem  that  they  are 
encountering.  Five  years  after  its  introduction,  the  Eureka  system  had  been  widely  adopted 
by  Xerox  technicians  and  has  resulted  in  significant  savings  in  time  and  parts  cost  (Bobrow 
&  Whalen,  2002). 

Modeling  and  Simulation  of  Collaborative  Systems 

The  SoS  modeling  and  simulation  approach  discussed  in  the  Introduction  section  is 
used  in  this  research  to  model  a  system  of  individual  system  programs  collaborating  to  form 
a  SoS.  This  M&S  approach  has  been  illustrated  with  eBay,  Facebook,  and  Eureka 
(Osmundson  &  Olgerson,  201 1 ).  To  be  self-contained,  this  paper  briefly  discusses  the  M&S 
approach  and  results  of  these  collaborative  systems. 

This  M&S  approach  considers  a  collaborative  system  to  consist  of  people,  databases 
and  other  elements.  People  interact  with  one  another  directly,  through  databases  and/or 
other  elements  to  achieve  outcomes.  The  collaborative  system  models  are  populated  with 
an  initial  population  of  users,  database  items,  and  other  necessary  elements.  Users  are 
assumed  to  want  to  match  with  other  users  or  database  items,  and  individual  user’s  desire 
to  join  the  system  and  remain  a  part  of  the  system  is  assumed  to  depend  on  their  success  in 
finding  matches.  Further,  the  probability  of  finding  a  match  is  assumed  to  depend  on  user 
and  database  item  populations,  the  type  of  collaborative  system,  and  the  number  and 
standard  deviations  of  the  parameters  that  are  required  to  determine  a  match.  People’s 
choices  are  heavily  influenced  by  other  people’s  choices.  Thus,  if  a  SoS  reaches  some 
critical  threshold  in  terms  of  number  of  users  and/or  number  of  successful  interactions — 
hits — one  would  expect  the  SoS  to  be  successful.  On  the  other  hand,  if  users  are 
unsuccessful  in  obtaining  useful  matches  with  the  system,  the  assumption  is  that  they  will 
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withdraw  from  the  system,  thereby  reducing  the  user  population  and  the  number  of  hits.  At 
some  point  in  time  the  population  should  keep  growing,  reach  a  stable,  useful  level,  or 
decline  to  a  point  where  the  system  is  no  longer  viable. 

The  probability  of  matches  may  depend  not  only  on  match  parameters  but  also  on 
the  manner  in  which  the  parameters  are  retrievable  by  the  users.  Each  model  of  a  specific 
type  of  system  has  one  or  more  probability  models  appropriate  to  the  type  of  system.  There 
can  also  be  competitive  behavior.  For  example,  users  may  want  recognition  for  having  the 
most  hits  on  their  blog  and  therefore  may  compete  with  other  users  in  creating  content. 


Figure  1 .  Abstract  Form  of  Web-Based  Systems  Model 

Three  types  of  discrete-event  models  represent  eBay,  Facebook,  and  the  Xerox 
Eureka  system.  Each  model  assumes  a  specific  type  of  attraction  mechanism,  unique  to 
each  system,  which  attracts  sufficient  users  over  time,  resulting  in  a  successful  system 
whose  value  exceeds  its  costs.  In  Figure  1 ,  the  users  interact  with  other  users  and/or  the 
web-based  system.  In  each  case  there  is  a  small  initial  seed  population  of  users.  If  the 
users  are  attracted  to  one  another  and/or  to  the  system  in  sufficient  numbers,  over  time  a 
successful  net  presence  ensues.  The  key  to  this  type  of  system  is  the  attractor  mechanism, 
which  is  the  mechanism  that  provides  value  to  the  users  while  at  the  same  time  a  cost  is 
imposed  on  the  users.  The  cost  could  be  a  monetary  fee  and/or — more  likely  in  many 
cases — the  time  and  effort  required  to  participate  in  the  system  and  the  potential  risk  in 
participating  in  the  system.  Each  of  the  models  of  the  three  types  of  systems  are 
implemented  in  Extend,®1  a  discrete-event  modeling  and  simulation  tool,  and  results  of  each 
of  the  three  types  of  models  agree  closely  with  real  world  data. 

Cost  and  value  are  specific  to  each  example  system.  As  the  eBay  model  represents 
on-line  sellers  and  buyers  of  a  variety  of  goods,  the  value  to  the  seller  is  low  cost  of  sales 
and  potentially  a  large  number  of  buyers  and  the  value  to  the  buyer  is  a  wide  selection  of 
goods  at  low  prices.  These  values  are  functions  of  the  number  of  users  over  time;  as  the 
number  of  sellers  and  buyers  increases  the  value  to  both  parties  increases.  There  are  also 
costs  to  the  seller  and  buyer.  The  seller  is  at  risk  of  not  being  paid  and  the  buyer  is  at  risk  of 
not  getting  the  goods  at  all  or  getting  miss-represented  goods  and/or  suffering  identify  theft. 
Initially  these  risks  were  relatively  high,  but  as  improvements  to  eBay  over  time,  such  as 
introduction  of  seller  ratings  and  use  of  Pay  Pal,  these  risks  declined.  Thus  the  value-to- 


1  Extend  is  a  product  of  Imagine  That  Inc.,  6830  Via  Del  Oro,  Suite  230,  San  Jose,  CA,  95119  USA. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-241  - 


cost  ratio  can  be  represented  by  the  time-dependent  number  of  eBay  users  and  an  S-curve 
function  representing  declining  risk  over  time.  The  rate  at  which  sellers  enter  the  system  is 
dependent  on  the  number  of  buyers  in  the  system.  The  buyers’  risk  factor  is  given  by  an  S- 
curve  function.  A  detailed  discussion  of  the  simulation  results  of  the  Extend  eBay  model,  as 
well  as  the  Extend  Facebook  model  and  the  Extend  Eureka  model,  is  in  Osmundson  and 
Holgerson  (201 1 ).  In  this  paper,  it  suffices  to  point  out  the  agreement,  as  shown  in  Figure  2, 
between  the  simulation  results  and  the  eBay  user  growth  data. 
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Figure  2.  Results  of  the  Extend  eBay  Model 

(“eBay. corn’s  Site  Profile,”  n.d.) 

Note.  Red  markers  are  model  results,  and  blue  markers  are  eBay  actual  growth  numbers 
(“eBay.com’s  Site  Profile,”  n.d.). 

The  Facebook  model  represents  people  who  want  to  form  social  networks  with  their 
friends.  The  value  to  each  individual  is  the  ability  to  communicate  on  a  regular  basis  with  a 
large  number  of  friends  by  posting  text  and  pictures  to  their  Facebook  homepage,  which  can 
be  viewed  by  their  friends.  Value  increases  with  the  number  of  friends  added  up  to  a  point 
where  the  cost  of  maintaining  meaningful  connections  is  outweighed  by  the  incremental 
value  of  adding  additional  friends  or  becoming  a  friend  on  another  person’s  site.  There  is  an 
initial  population  of  participants  and  new  participants  arrive  at  a  rate  proportional  to  the  total 
population.  Participants  look  for  a  match — that  is,  a  friend,  and  the  probability  of  finding  a 
friend  is  proportional  to  the  total  population.  As  shown  in  Figure  3,  the  Extend  model  results 
fit  the  actual  Facebook  population  data  fairly  well  through  the  first  41  months,  but  beyond 
that  point  the  model  population  grows  at  a  rate  faster  than  the  actual  population.  The 
Extend  model  is  a  very  simple  model  and  does  not  include  any  saturation  effects  such  as 
might  occur  if  the  early  adopters  of  Facebook  are  more  likely  to  find  friends  among  a  given 
population  than  are  late  arrivals,  or  if  the  Facebook  population  begins  to  approach  a  limit  of 
all  possible  networked  users. 
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Figure  3.  Results  of  the  Extend  Facebook  Model 

A/ofe.  Red  markers  are  model  results  and  blue  markers  are  Facebook  actual  growth  numbers 

(Statistics,  n.d.). 

The  Eureka  model  begins  with  generation  of  experts  who  initially  are  assigned 
problems  randomly;  the  experts  then  enter  tips  for  solving  each  of  the  problems.  This 
generates  an  initial  set  of  validated  tips.  Other  technicians  are  generated  next.  The  experts 
are  randomly  assigned  new  problems,  they  check  the  data  base  for  tips  and  if  a  tip  exists 
they  utilize  it  and  solve  the  problem  quickly.  If  no  tip  exists  they  take  a  long  time  to  solve  the 
problem  and,  with  some  probability,  enter  either  a  new  tip  or  not.  The  probability  of  entering 
a  new  tip  is  given  by  an  S-curve  function  that  is  dependent  on  the  number  of  times  a  given 
person’s  tips  have  been  utilized.  This  reflects  the  fact  that  technical  workers  are  highly 
motivated  by  peer  recognition  and  is  consistent  with  Xerox’s  experience. 

The  Eureka  model  was  initially  run  and  the  probability  with  which  technicians 
checked  the  database  was  adjusted  until  a  best  fit  was  obtained  with  real  world  data.  The 
best  fit  occurred  when  the  probability  of  checking  the  database  at  a  given  time  was  set  to 
0.4T/P,  where  T  is  the  number  of  tips  generated  up  to  a  given  time  and  P  is  the  total  number 
of  problems  that  are  expected  to  be  encountered.  Based  on  available  data  on  Xerox’s 
Eureka  system,  the  initial  number  of  tips  was  100-200  (“The  Eureka  Project,”  2010),  the 
total  number  of  technicians  during  the  first  5  years  of  use  was  19,000,  the  number  of 
technicians  participating  in  the  Eureka  system  after  5  years  was  15,000  and  the  total 
number  of  unique  vetted  tips  after  5  years  was  36,000.  The  total  number  of  problems  to  be 
solved  was  not  available;  for  purposes  of  calibrating  the  model  the  total  number  of  problems 
was  assumed  to  be  50,000.  It  was  also  assumed  that  it  took  an  average  of  1  hour  to  solve  a 
problem  with  a  tip  and  an  average  of  8  hours  without  a  tip  and  that  technicians  completed 
approximately  one  trouble  call  per  day.  Jack  Whalen  (personal  communication,  October  14, 
2010)  estimated  that  non-routine  problems  occurred  more  frequently  than  once  per  week, 
but  less  frequently  than  once  per  day. 

The  most  important  measure  of  effectiveness  of  this  type  of  system  is  the 
participation  rate.  The  participation  rate  drives  the  number  of  new  tips  generated  over  time 
and  is  the  main  factor  in  determining  the  reduction  in  time  to  solve  problems.  Participation 
rates  at  the  end  of  one  year  and  at  the  end  of  five  years,  as  a  function  of  initial  tips  and  total 
expected  problems,  are  shown  in  Figures  4  and  5,  respectively.  The  results  clearly  show 
that  the  ratio  of  initial  tips  to  number  of  expected  problems  to  be  encountered  is  critical  to 
success,  particularly  in  achieving  a  reasonably  high  rate  of  technician  participation. 
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Figure  4.  Participation  Rate  After  One  Year  as  a  Function  of  Initial  Tips  and 
Number  of  Problems  to  be  Encountered 


Figure  5.  Participation  Rate  After  Five  Years  as  a  Function  of  Initial  Tips  and 
Number  of  Problems  to  be  Encountered 

SoS  Inter-Program  Collaboration  Approach 

As  discussed  in  the  Introduction  section,  collaboration  among  the  individual  system 
programs  participating  in  a  SoS  acquisition  depends  on  the  presence  of  mechanisms  to 
induce  the  willingness  on  the  part  of  the  individual  programs  to  collaborate  and  to  enable 
their  collaboration.  Mechanisms  can  include  formalized  structures  for  coordination; 
formalized  processes  including  meetings,  deadlines,  etc.;  sufficient  authority  of  participants; 
clarity  of  roles;  and  assets  such  as  personnel  that  are  dedicated  for  collaboration.  Lateral 
mechanisms  can  include  interpersonal  networks,  effective  communication  and  information 
exchange,  technical  interoperability,  and  training  (Hocevar  et  al.,  2006).  As  discussed  in  the 
section  on  Modeling  and  Simulation  of  Collaborative  Systems,  a  web-based  service-oriented 
architecture  is  an  efficient  means  of  providing  a  mechanism  that  provides  many  of  the 
mechanistic  requirements  for  collaboration.  However,  successful  web-based  collaboration  is 
highly  dependent  on  the  value/cost  ratio  that  applies  to  a  given  system. 

Like  eBay,  Facebook,  and  Eureka,  the  collaborative  system  envisioned  for  SoS 
acquisition  needs  to  have  an  attraction  mechanism — to  attract  the  individual  programs  to 
collaborating  with  the  other  programs  to  achieve  the  objectives  of  the  SoS  acquisition 
program.  Such  a  mechanism,  just  like  those  implemented  with  eBay,  Facebook,  and 
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Eureka,  should  be  highly  related  cost  and  value  of  collaboration,  as  it  provides  value  to  the 
participating  programs  while  at  the  same  time  a  cost  is  imposed  on  them. 

Each  individual  program  invariably  is  burdened  with  the  production  of  a  system  with 
required  performance  on  schedule  and  within  budget.  Consequently,  the  value  and  cost 
derived  from  collaborating  with  the  other  programs  are  related  to  these  parameters — 
performance,  schedule,  and  budget.  There  is  also,  however,  another  element  that  can 
highly  motivate  participation  in  a  collaborative  system — recognition.  Value  is  in  terms  of 
recognition.  In  the  Eureka  system,  if  technicians  see  that  another  worker  has  been 
recognized  for  providing  a  tip  for  solving  repair  problems,  they  too  will  want  similar 
recognition  and  will  be  motivated  to  enter  a  new  tip.  If  a  technician  sees  that  his  own  tip  has 
been  useful  to  others,  he  will  be  motivated  to  provide  additional  tips  in  order  to  achieve 
further  peer  recognition.  Thus,  in  addition  to  promoting  value  and  compensating  for  cost, 
recognition  should  be  instituted  for  contributing  to  the  development  of  the  SoS  acquisition. 
But,  in  what  form  should  recognition  be  realized — money,  promotion,  reputation,  rewards 
beyond  a  program  manager’s  tour  on  the  program?  And  to  whom  should  recognition  be 
attributed — just  to  the  program  managers,  or  to  the  entire  team? 

Some  contributors  to  the  cost  of  collaboration,  hence  the  barriers  to  collaboration, 
are  observed.  The  cost  of  dedicating  their  resources  to  developing  the  parts  that  are 
required  to  satisfy  the  SoS  requirements.  The  cost  to  program  personnel  collaborating  in 
this  effort  is  the  additional  time  spent  on  executing  the  SoS  part  of  the  system.  The  cost 
associated  with  a  potential  delay  in  the  development  of  their  own  systems,  caused  by  their 
participation  in  the  SoS  development.  The  individual  programs  that  are  not  compensated  for 
these  costs  will  more  than  likely  decline  to  participate  or  pay  lip  service  to  collaborating  in 
the  SoS  acquisition. 

Assessing  value  for  collaborating  is  more  problematic.  There  is  high  value  to  the 
overall  SoS  through  keeping  the  individual  system  programs  aligned  in  order  to  support  SoS 
testing,  but  there  is  not  necessarily  much  value  to  each  individual  component  program. 
Program  managers  are  typically  rewarded  for  producing  the  desired  system,  on  time  and 
within  budget,  but  they  are  not  presently  rewarded  for  aligning  their  programs  with  other 
programs.  Value  to  individual  program  managers  and  program  offices  must  be  provided  in 
order  to  achieve  effective  collaboration. 

The  system  factors  that  determine  the  success  or  failure  of  this  collaborative  SoS 
acquisition  system  include  the  number  of  participating  programs  which  depend  on  the 
aforementioned  incentives,  the  factors  that  attract  a  collaborator,  the  factors  that  cause  an 
individual  program  to  continue  to  buy  in  collaboration,  the  values  of  the  SoS  to  the 
participating  programs,  and  the  cost  or  risks  to  their  programs.  As  in  the  web-based 
collaborative  systems  discussed  above,  it  is  assumed  that  a  participant  wants  to  find  a 
“match”  in  the  collaborative  system.  That  match  need  be  understood  for  the  collaborative 
SoS  acquisition  system.  Value  to  the  participant  is  associated  with  finding  a  match. 

One  mechanism  that  holds  promise  for  meeting  many  of  the  requirements  for  inter¬ 
program  collaboration  is  a  web-based  service-oriented  architecture  system  on  which 
personnel  of  all  programs  associated  with  a  SoS  can  input  and  retrieve  information  required 
to  align  the  individual  programs: 

Service  Oriented  Architecture  (SOA)  is  an  architectural  paradigm  and  discipline 
that  may  be  used  to  build  infrastructures  enabling  those  with  needs  (consumers) 
and  those  with  capabilities  (providers)  to  interact  via  services  across  disparate 
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domains  of  technology  and  ownership.  Services  act  as  the  core  facilitator  of 
electronic  data  interchanges  yet  require  additional  mechanisms  in  order  to 
function.  Several  new  trends  in  the  computer  industry  rely  upon  SOA  as  the 
enabling  foundation,  (www.adobe.com) 

As  discussed  in  the  Introduction  section,  to  achieve  a  successful  SoS  acquisition,  we 
propose  a  web-based  service-oriented  architecture  on  which  personnel  of  all  programs 
associated  with  a  SoS  can  input  and  retrieve  information  required  to  align  the  individual 
programs. 

The  overall  development  of  the  SoS  and  component  systems  is  treated  as  a  critical- 
path  network  and  the  need  points  for  component  system  inputs  are  identified  as 
intermediate  milestones  requiring  SoS-component  system  collaboration,  typically  a  joint 
review.  This  approach  is  consistent  with  knowledge-based  acquisition,  since  the  SoS 
development  proceeds  only  as  the  required  component  information  is  available.  In  this  work 
we  analyze  the  development  of  a  SoS  from  a  systems  engineering  perspective,  identifying 
the  points  in  the  SoS  development  where  information,  surrogates,  software  and  hardware 
are  needed  from  the  component  systems.  The  web-based  SoS  acquisition  system  is 
envisioned  to  incorporate  the  knowledge-based  acquisition  approach. 

An  Extend  model  is  built  to  capture  this  web-based  service-oriented  architecture.  An 
attraction  mechanism  is  incorporated  in  the  model.  Simulation  of  this  model  then  provides 
results  to  establish  the  feasibility  of  such  a  SoS  collaborative  system. 

Conclusion 

System-of-systems  (SoS)  acquisition  research  has  identified  lack  of  alignment  and 
lack  of  collaboration  as  two  important  issues  leading  to  problems  in  SoS  acquisition.  This 
paper  captures  the  exploratory  work  toward  improving  alignment  between  and  collaboration 
among  the  individual  system  programs  in  the  development  of  a  SoS.  Inspired  by  some 
existing  web-based  collaborative  systems,  such  as  eBay,  Facebook,  and  Eureka,  a 
collaborative  web-based  system  is  proposed,  on  which  personnel  of  all  programs  associated 
with  a  SoS  can  input  and  retrieve  information  required  to  align  the  individual  programs. 

The  overall  development  of  the  SoS  and  component  systems  is  treated  as  a  critical- 
path  network  and  the  need  points  for  component  system  inputs  are  identified  as 
intermediate  milestones  requiring  SoS-component  system  collaboration.  An  attraction 
mechanism  to  effect  SoS  inter-program  collaboration  is  incorporated  in  a  model  capturing 
this  web-based  SoS  collaborative  system.  Simulation  using  this  model  then  provides  results 
to  establish  the  feasibility  of  such  a  SoS  collaborative  system. 

This  work  forms  a  basis  for  building  a  web-based  SoS  collaborative  system  to 
support  DoD  SoS  acquisition  programs. 
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Summary  of  Preliminary  Research 


•  Previous  System-of-systems  (SoS)  acquisition  research  identified 
two  important  issues  leading  to  problems  in  SoS  acquisition: 

-  lack  of  alignment 

-  lack  of  collaboration. 


•  A  collaborative  web-based  system  is  proposed 

-  Personnel  of  all  SoS  associated  programs  can  input  and  retrieve 
information  required  to  align  individual  programs. 

•  Development  of  the  SoS  and  component  systems  is  treated  as  a 
critical-path  network 

-  Need  points  for  SoS-component  collaboration  component  are  identified 

•  Successful  collaborative  web-based  systems  have  been  analyzed 
and  a  success  factor  has  been  identified 

•  An  attraction  mechanism  to  effect  SoS  inter-program  collaboration 
has  been  characterized 
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Lack  of  Alignment  Issues 


•  Most  common  type  of  DoD  SoS  development  is  one  in  which  a  SoS  is  to  be 
created  by  integrating  separately  developed  systems  -  legacy  systems, 
developmental  systems,  or  some  combination  of  both 

•  Lack  of  alignment  means  a  system  is  not  ready  for  its  integration  into  a  SoS 
,  a  result  of 

-  Lack  of  the  front-end  SoS  systems  engineering  (SE) 

-  And/or  lack  of  collaboration 

•  SoS  integration  testing  requires  the  availability  of  surrogates  and  later  the 
“as  built”  component  systems  in  a  timely  manner 

-  Acquisition  schedules  for  the  component  systems  are  typically 
developed  independently  of  the  SoS  development  schedule. 

-  No  assurance  that  the  SoS  integration  testing  can  be  completed  as 
planned 

-  Even  when  the  schedules  are  aligned  a  component  system  may  not 
meet  the  performance  or  interface  requirements 


Lack  of  Collaboration  Issues 


•  The  lack  of  alignment  is  related  to  the  lack  of 
collaboration. 

•  Collaboration  in  the  development  of  a  SoS  is  multi¬ 
dimensional 

-  Between  DoD  system  program  offices 

-  Between  contractors 

-  Between  DoD  program  offices  and  contractors 

•  Inter-organizational  collaboration  requires  collaborative 
capacity  -  the  ability  of  individual  system  programs  to 
enter  into,  develop,  and  sustain  inter-system  programs  in 
the  pursuit  of  SoS  collective  outcomes 


Issues  in  Achieving  Effective  SoS 
Collaboration 


•  Individual  system  programs  must  have  the  ability  to  enter  into, 
develop,  and  sustain  inter-systems  programs  collaboration 

-  There  must  be  a  mechanism  of  to  develop  and  maintain  such  an 
ability. 

•  Mechanisms  include  structures  for  coordination 


-  Meetings,  deadlines,  etc.;  sufficient  authority  of  participants;  clarity 
of  roles;  and  assets  such  as  personnel  that  are  dedicated  for 
collaboration 


-  Interpersonal  networks,  effective  communication  and  information 
exchange,  technical  interoperability,  and  training  (Hocevar  et  al. 
2006) 


•  Mechanisms  remove  barriers  against  and  implement  factors 
favorable  to  the  realization  of  collaborations 

•  However,  there  is  cost  associated  with  implementing  and 
maintaining  collaboration  mechanisms 


Hocevar,  S.P.,  Thomas,  G.F.,  &  Jansen,  E.,  “Building  collaborative  capacity:  An  innovative  strategy  for  homeland  security  preparedness,”  in 
Beyerlein,  Beyerlein  &  Kennedy  (Eds.),  Advances  in  interdisciplinary  studies  of  work  teams:  Innovations  through  collaboration,  pp.  263- 
283,  Vol.  12,  New  York:  Elsevier  JAI  Press,  2006. 
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Web-Based  Collaboration  Mechanisms 


•  In  this  work,  the  key  idea  is  to  capitalize  on  collaborative 
behavior  observed  on  some  existing  web-based  systems 

-  Web-based  systems  minimize  the  cost  of  face-to-face  meetings  yet 
meet  collaboration  requirements  including  support  of  interpersonal 
networks,  effective  communication,  ease  of  training,  ... 


•  The  web-based  system  concept  inspires  the  mechanism 
proposed  in  this  research  for  inter-program  collaboration 

•  A  collaborative  web-based  system  is  proposed,  on  which 
personnel  of  all  programs  associated  with  a  SoS  can  input 
and  retrieve  information  required  to  align  the  individual 
programs 

•  To  identify  and  assess  the  issues  associated  with  achieving 
web-based  inter-program  collaboration,  modeling  and 
simulation  (M&S)  is  employed 
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Modeling  and  Simulation  of  Successful 
Web-Based  Collaborative  Systems 


•  What  are  the  factors  that  lead  to  successful  collaboration 
in  on-line  systems? 


•  Successful  on-line  systems  have  been  analyzed  by 
modeling  them  as  SoSs  consisting  of  users,  data  bases, 
computers  and  networks 
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System 


Usere 


Three  Successful  Web-Based  Systems 
Have  Been  Analyzed  Using  M&S 


•  eBay,  Facebook  &  Xerox’s  Eureka 

•  Each  was  modeled  using  the  discrete  event 
simulation  application  Extend™ 


Attractor  mechanisms  are  key  to  achieving  a 
sustainable,  high  participation  rate 

Each  model  included  a  function  of  cost  and/or 
value  that  influenced  the  attractor  mechanism 

Cost  and/or  value  functions  all  had  the  form  of  S 
curves 
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eBay 


•  eBay  is  an  on-line  auction  and  shopping  website 

•  Value  to  the  seller  is  low  cost  of  sales  and  potentially  a  large  number 
of  buyers 

•  Value  to  the  buyer  is  a  wide  selection  of  goods  at  low  prices 

•  One  of  the  important  costs  is  risk,  both  to  the  buyer  and  seller 

-  Buyer  may  not  get  goods  or  the  goods  may  be  misrepresented 

-  Seller  may  not  get  paid 


•  Initially  the  risk  was  high  but  decreased  with  time  as  seller  ratings 
and  secure  payment  methods  were  introduced 

•  Risk  =1/(1  +  e<t/2-t)),  an  S-shaped  function  where  t  is  the  time  after 
the  start  of  eBay 
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eBay  Model  Results 
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Facebook 


•  Facebook  facilitates  people  forming  social  networks 
with  their  friends 


•  Value  to  participants  is  ability  to  communicate  on  a 
regular  basis  with  a  large  number  of  friends  by 
posting  text  and  pictures 


•  Cost  is  the  time  and  effort  to  maintain  one’s  network 
of  friends. 


•  Cost  =1/(1+  e(N/2-N>),  an  S-shaped  function  where 
N  is  the  number  of  friends  in  a  network  and  N  =  130 
for  the  average  Facebook  participant 
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Facebook  Model  Results 
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Xerox’s  Eureka  System 


•  Eureka  allows  service  engineers  to  search  on-line  for  solutions  to  copier 
miss-usage  and  repair  problems 

•  A  few  expert  service  engineers  initially  populated  the  system  with  -200 
validated  problem  solving  tips 

•  Other  technicians  began  to  utilize  the  database  of  validated  tips  and 
then  occasionally  would  enter  new  tips 


•  Interestingly  -  even  though  the  system  allowed  service  engineers  to 
save  time  -  the  main  value  to  service  engineers  seems  to  be  peer 
recognition  received  when  credited  with  a  useful  tip 

•  Recognition  =  1/(1  +  e(T/2-T)),  an  S-shaped  function  where  T  is  the 
number  of  tips  credited  to  an  individual  and  T  =  5  for  the  average 
Eureka  service  engineer.  It  seems  that  at  T  =  5  the  additional 
recognition  was  not  worth  the  additional  effort  to  enter  a  new  tip. 
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Calibration  of  the  Eureka  Model  to  Best 
Fit  Xerox  Data 


Parameter 

Eureka 

Extend  Model 

Model  Result 

Expected  Result 
Based  on  Scaling 

Initial  tips 

200 

10 

Technicians 

19,000 

1000 

Problems 

50,000 

(estimated) 

2500 

Tips 

36,000  @  5  years 

1 820  @  5  years 

1800  @  5  years 

Participation  rate 

79% 

65%  @  5  years 

79%  @  5  years 

Reduction  in 
time  to  solve 
problems 

-10% 

-10% 

-10% 
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Successful  Web-Based  Systems  and  the 
Challenge  to  SoS  Acquisition 


•  Successful  on-line  collaborative  systems  have  attractor  mechanisms 
that  are  governed  by  a  positive  value/cost  attractor  mechanism 

•  The  challenge  in  SoS  acquisition  is  to  develop  an  attractor 
mechanism  that  provides  positive  value  to  individual  SoS  programs 

•  There  is  value  to  the  overall  SoS  program  office  at  minimal  cost 

-  Better  control  of  cost  and  schedule 


-  Higher  assurance  of  meeting  performance  goals 

•  There  is  higher  cost  to  the  component  system  program  offices 

-  Investment  in  implementing  the  system  and  training 

-  More  time  spent  in  the  collaborative  process 


•  What  is  the  value  to  the  individual  program  offices? 

-  Typically  PMs  are  judged  and  rewarded  on  the  basis  of  meeting  their  own 
program’s  cost,  schedule  and  performance  goals,  not  the  SoSs’ 

•  The  challenge  is  to  provide  sufficient  value  to  individual  programs  in 
an  on-line  collaborative  system 
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Conclusion 


•  System-of-systems  (SoS)  acquisition  research  has 
identified  lack  of  alignment  and  lack  of  collaboration  as 
two  important  issues  leading  to  problems  in  SoS 
acquisition 

•  A  collaborative  web-based  system  is  proposed,  on  which 
personnel  of  all  programs  associated  with  a  SoS  can 
input  and  retrieve  information  required  to  align  the 
individual  programs 

•  An  attraction  mechanism  to  effect  SoS  inter-program 
collaboration  must  be  identified 

•  This  work  will  form  a  basis  for  building  a  web-based  SoS 
collaborative  system  to  support  DoD  SoS  acquisition 
programs 
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